Can you add a test for when NaN is somewhere in a Vector and you try to sort it? That was a requirement by @jdunkerley IIRC

This will be tackled in #5742, in a follow-up PR.

I'm hesitant of expanding our numeric type hierarchy like that,

+1 - Do Not Expose Deep Hierarchies, Practical API Design page 83

it feels like it will drive up the complexity of the type system

Do you mean implementation? No, that's OK.

I should probably note that the core flaw is our "attempt to not support partial order on any type". If our comparator method could return:

compare :: T -> T -> Ordering | Nothing

we wouldn't have this problem. I am OK with the introduction of Nan type, but I would much rather see us supporting "partial ordering" by default. E.g. return the design back where I left it:

• Custom eq & hash with consistency #5008

Then we wouldn't need to create an artificial type like Nan.

I should probably note that the core flaw is our "attempt to not support partial order on any type". If our comparator method could return:
compare :: T -> T -> Ordering | Nothing
we wouldn't have this problem. I am OK with the introduction of Nan type, but I would much rather see us supporting "partial ordering" by default. E.g. return the design back where I left it:
#5008
Then we wouldn't need to create an artificial type like Nan.

Let's stick with the current design for now and see where it takes us. I assume that we will know for sure during the work on #5626. Potential rollback to compare: T -> T -> (Ordering | Nothing) should not be that difficult. After all, even stdlib has not fully migrated yet.